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Abstract 


Acquiring  software-reliant  systems  from  an  external  resource  can  be  time  consuming,  costly,  and 
often  unreliable.  During  independent  technical  assessments  and  customer  engagements,  the  Soft¬ 
ware  Engineering  Institute  observed  many  contractors  who  are  not  utilizing  mature,  readily  avail¬ 
able  software  development  tools  when  creating  code  for  govermnent  programs.  These  tools  in¬ 
clude  static  analysis,  test  automation,  and  peer  review  techniques.  In  addition  to  the 
aforementioned  tools  being  important  for  software  developers,  they  offer  significant  insight  and 
confidence  to  customers  who  acquire  software  products. 

There  are  many  benefits  from  making  these  tools  and  practices  integral  to  software  development 
and  acquisition  processes,  including: 

•  reduced  risk:  programs  deliver  more  predictably 

•  improved  customer  satisfaction:  products  developed  experience  fewer  field  defects 

•  lower  cost  of  ownership:  programs  experience  lower  life-cycle  maintenance  costs  when  de¬ 
ployed  operationally 

This  technical  note  provides  an  introduction  to  key  automation  and  analysis  techniques,  the  use  of 
which  the  authors  contend  will  benefit  motivated  acquirers  and  developers. 
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1  State  of  Software  Acquisition  and  Development  in  Large 
Systems 


Increasing  demands  for  more  functionality  alongside  improving  important  system  qualities  like 
reliability,  availability,  and  security  are  driving  up  the  complexity  of  developing  software-reliant 
systems.  Continually  striving  to  reduce  cost  and  schedule  is  a  key  requirement  for  surviving  in  a 
competitive  market;  often,  the  result  is  that  low-quality  code  is  rushed  to  production.  For  large 
mission-critical  systems  such  as  military  and  banking  applications,  significant  delays  may  occur 
and  not  be  fully  realized  until  late  in  the  program  development  cycle.  Although  it  is  not  usually 
obvious,  one  of  the  more  effective  actions  to  improve  cost  and  schedule  is  to  improve  efficiency 
of  code  development.  By  doing  this,  more  time  can  be  spent  on  design  and  test  efforts  that  further 
support  the  overall  quality  of  the  delivered  system. 

A  considerable  number  of  robust  and  reliable  software  development  tools  and  practices  have  been 
developed  over  the  years  and  have  been  accepted  as  state  of  the  practice,  enabling  developers  to 
deliver  quality  products  on  a  predictable  schedule.  The  open-source  community  has  created  inex¬ 
pensive  and  effective  tools.  In  addition,  many  are  not  complicated  and  with  minimal  training  can 
be  used  across  an  organization’s  development  process.  These  tools  and  practices  also  become  crit¬ 
ical  to  the  success  of  the  use  of  new  methodologies  such  as  agile  software  development.  There¬ 
fore,  it  is  not  clear  to  the  authors  why  many  acquisition  and  software  development  programs  we 
have  encountered  do  not  incorporate  the  use  of  these  tools  into  their  software  development  pro¬ 
cesses. 

This  section  explores  the  current  state  of  software  acquisition  and  development  with  a  focus  on 
challenges  related  to  large  system  development.  Section  2  describes  recommended  tools  for  every 
software  development  project.  The  Section  3  looks  at  automation,  peer  reviews,  and  some  select¬ 
ed  management  processes  that  have  tools  and  practices  applicable  to  most  phases  of  software  de¬ 
velopment.  Section  4  concludes  with  recommendations  as  to  how  acquisition  organizations 
should  apply  these  development  tools  and  processes. 

1.1  Increasing  Quality  and  Development  Efficiency 

Creating  software  is  a  complex  process  that  requires  large  numbers  of  people  with  a  diverse  range 
of  technical  skills.  Making  improvements  to  the  software  development  process  has  been  a  chal¬ 
lenge  for  many  years,  but  throughout  the  short  history  of  software  development,  a  wide  range  of 
improvements  have  allowed  the  industry  to  increase  the  functionality,  quality,  and  development 
efficiency  of  delivered  systems.  This  list  includes  highly  structured  approaches  for  process  im¬ 
provement  like  CMMI®  and  software  development  processes  like  Rational  Unified  Process 
(RUP).  Somewhat  orthogonal  to  these  structured  methods  are  software  development  models  such 
as  waterfall,  spiral,  and  agile.  One  can  also  look  at  the  various  design  modeling  approaches  such 
as  structured  decomposition,  object  oriented,  aspect  oriented,  and  others.  Additionally,  over  the 
past  25  years  or  so,  a  focus  has  been  on  software  architecture  as  an  impactful  way  to  identify  im¬ 
portant  quality  attributes  (QAs)  early  in  the  design  process.  The  architecture  area  also  included  a 
focus  on  architectural  and  more  detailed  design  patterns  as  a  way  of  addressing  common  issues 
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found  during  software  development.  These  and  other  ideas  were  directed  toward  addressing  the 
challenge  of  how  to  best  deliver  quality  code  on  a  predictable  schedule  and  cost. 

Increased  quality  earlier  in  the  software  development  process  provides  increased  productivity,  and 
software  development  life  cycles  often  become  more  predictable  and  thus  more  manageable,  re¬ 
sulting  in  lower  costs.  In  both  commercial  and  government  acquisitions,  the  quality  of  the  devel¬ 
oped  software  is  a  key  factor  to  the  overall  cost  and  schedule  of  the  delivered  software  system  and 
software  sustainability  throughout  the  lifecycle.  Effective  software  development  processes,  utiliz¬ 
ing  accepted  software  engineering  practices  and  tools,  make  for  more  predictable  software  devel¬ 
opment. 

During  the  development  process  there  are  many  opportunities  for  the  injection  of  defects  into  the 
resulting  code.  A  typical  software  development  process  usually  consists  of  these  general  activi¬ 
ties: 


•  requirements  engineering 

•  architecture  and  detailed  design 

•  code  and  unit  test 

•  integration 

•  system  test 

Many  of  today’s  development  methodologies  encourage  the  above  processes  to  occur  simultane¬ 
ously,  often  overlapped  in  time  to  some  extent,  and  continuously  throughout  the  life  cycle  of  the 
product. 

A  basic  tenet  of  creating  quality  software  is  to  detect  and  correct  defects  close  to  when  they  are 
introduced.  Boehm  and  Papaccio  reported  in  1988  that  requirements  defects  found  after  a  system 
is  fielded  cost  50  to  200  times  more  to  rectify  than  if  they  had  been  found  and  corrected  near  the 
beginning  of  the  development  [Boehm  1988]. 

The  current  situation  is  not  much  different  today  as  industry  data  still  supports  the  notion  that  er¬ 
rors  in  requirements  specification  are  expensive  to  fix:  Wiegers  asserts  that  50  percent  of  defects 
originate  in  the  requirements  phase,  and  80  percent  of  the  rework  cost  on  a  project  can  be  traced 
to  defects  arising  during  the  requirements  specification  phase  [Wiegers  2001],  Granted  that  pre¬ 
venting  defects  in  the  early  phases  like  requirements  and  design  give  the  greatest  payback,  this 
paper  examines  tools  and  processes  that  help  to  find  defects  injected  during  the  implementation 
phase.  Quite  often  these  defects  are  not  found  until  the  integration  and  system  test  phases.  When 
the  tools  and  processes  recommended  in  this  paper  are  used  throughout  the  implementation  phase, 
they  can  identify  many  defects  that  are  relatively  easy  to  find  and  fix. 

1.2  Acquisition  and  Development  Challenges 

Many  large  development  projects  today  involve  an  acquisition  team  (acquirer)  working  with  a 
development  team  (developer)  to  create  and  deliver  software-intensive  systems.  While  the  acqui¬ 
sition  team  usually  focuses  on  the  financial,  business,  and  political  aspects,  they  also  need  visibil¬ 
ity  into  the  technical  development  status  and  progress,  and  the  quality  of  the  developing  system. 
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In  the  United  States  Department  of  Defense  (DoD),  acquisition  of  software-intensive  solutions  is 
primarily  performed  through  the  use  of  program  offices  contracting  with  defense  contractors  to 
define,  develop,  and  deliver  software-intensive  systems.  In  the  commercial  world,  acquisition  is 
usually  done  through  business  and  marketing  teams  working  closely  with  internal  or  external  de¬ 
velopers  to  deliver  the  desired  products.  In  both  realms,  the  software  development  processes  are 
similar  so  these  tools  and  practices  are  applicable  in  both  domains. 

Govermnent  and  commercial  acquisitions  have  historically  encouraged  the  use  of  process  models 
and  frameworks  like  CMM/CMMI,  ISO9001,  and  Information  Technology  Infrastructure  Library 
(ITIL)  for  many  years  in  an  attempt  to  gain  confidence  that  the  development  team  can  consistently 
deliver  as  promised.  Likewise,  process  methodologies  like  Total  Quality  Management  (TQM), 

Six  Sigma,  and  Lean  have  been  part  of  the  software  engineering  culture  for  many  years.  These 
models  and  methodologies  are  accepted  as  good  practices,  yet  they  have  been  slow  to  be  adopted. 
While  the  authors  firmly  advocate  the  use  of  the  above  models  and  methodologies,  this  paper  ar¬ 
gues  that  acquirers  can  make  some  basic  improvements  to  their  oversight  solely  by  using  state-of- 
the-art  software  engineering  tools  and  practices. 

The  CMMI-ACQ  model  (Capability  Maturity  Model-Integrated  for  Acquisition)  discusses  objec¬ 
tives  that  an  acquirer  should  actively  manage.  The  introduction  to  the  CMMI-ACQ  states:  “The 
acquirer  owns  the  project,  executes  overall  project  management,  and  is  accountable  for  delivering 
the  product  or  service  to  the  end  users.  Too  often  acquirers  disengage  from  the  project  once  the 
supplier  is  hired.  Too  late  they  discover  that  the  project  is  not  on  schedule,  deadlines  will  not  be 
met,  the  technology  selected  is  not  viable,  and  the  project  has  failed”  [CMMI  Product  Team 
2010], 

In  addition,  the  acquisition  technical  management  process  area  from  CMMI-ACQ  focus  the  ac¬ 
quirer  on  the  following  activities  [CMMI  Product  Team  2010]: 

•  conducting  technical  reviews  of  the  supplier’s  technical  solution 

•  analyzing  the  development  and  implementation  of  the  supplier’s  technical  solution  to  confirm 
technical  progress  criteria  or  contractual  requirements  are  satisfied 

•  managing  selected  interfaces 

There  are  varied  strategies,  processes,  and  methods  that  the  acquirer  may  leverage  to  develop  an 
accurate  understanding  of  developer  progress,  technical  decisions,  and  the  quality  of  the  delivered 
product.  They  range  from  using  fonnal,  documentation-driven  processes  to  including  the  acquirer 
as  an  active  participant  in  the  decision-making  process.  The  more  confidence  that  the  acquirer  has 
in  the  contractor’s  process,  the  less  friction  there  will  be  between  the  two  parties. 

1.3  Acquisition  and  Development  Process  Improvement 

Typically,  acquisition  organizations  focus  on  schedule  and  cost  and,  as  Watts  Humphrey  ob¬ 
served, 

If  the  organization  is  not  cost  competitive,  or  if  it  produces  lower  quality  or  less  attractive 
products,  a  focus  on  current  performance  will  not  improve  the  situation.  The  immediate 
problems  may  be  fixed  and  the  burning  issues  resolved,  but  the  organization  will  continue 
working  pretty  much  as  it  always  has.  It  will  thus  continue  producing  essentially  the  same 
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results  and  generating  essentially  the  same  problems  and  issues.  This  brings  us  to  the  defini¬ 
tion  of  insanity:  doing  the  same  thing  over  and  over  and  expecting  a  different  result  [Humph¬ 
rey  2009], 

A  key  reason  to  improve  productivity  through  the  incorporation  of  software  development  tools 
and  practices  is  to  free  up  engineering  talent  from  non-value-added  activities.  It  is  also  a  way  to 
improve  schedule  and  cost  predictability.  The  large  body  of  work  in  process  improvement  and 
change  management  demonstrates  that  successful  organizations  continually  reflect  on  the  current 
process  and  identifying  opportunities  for  improvement.  In  almost  any  process  improvement  meth¬ 
odology,  a  key  focus  is  on  avoiding  rework.  Since  humans,  and  thus  software  engineers,  are  not 
perfect,  finding  ways  to  detect  and  remove  defects  early  is  an  important  part  of  all  software  devel¬ 
opment.  As  discussed  in  section  1 . 1 ,  it  is  generally  less  costly  to  detect  and  correct  an  error  closer 
to  the  time  when  it  was  introduced. 

The  processes  used  by  the  developer  to  create  the  software  directly  affects  the  quality  of  the  deliv¬ 
ered  software  product.  Mature  software  development  tools  and  practices  can  give  the  acquirer 
important  indicators  and  predictors  of  quality.  A  key  challenge  from  the  acquirer’s  point  of  view 
is  gaining  insight  into  the  progress  on  and  quality  of  the  software  being  developed.  Development 
of  this  insight  can  take  many  forms  including 

•  observation  of  progress  measures  at  key  milestone  events 

•  review  of  process  artifacts  to  gain  insight  as  to  how  closely  the  developer  followed  the  de¬ 
fined  processes 

•  review  of  intermediate  work  products  and  metrics 

•  participation  in  the  actual  development  processes  through  active  review  of  code  and  artifacts 

•  side-by-side  participation  of  acquisition  personnel  supplying  real-time  requirements,  peer 
review,  and  verification 

The  objective  of  process  improvement  efforts  is  to  increase  the  effectiveness  of  system  and  soft¬ 
ware  development  talent.  The  end  game  is  about  producing  software  intensive  systems  that  are 
high  quality,  within  budget  and  schedule.  The  authors  advocate  the  redirection  of  engineering  ef¬ 
forts  to  be  focused  on  higher  value  activities  such  as 

•  focus  on  deeper  understanding  of  the  written  and  implied  requirements 

•  create  and  tune  the  architecture  and  detailed  design  of  the  system  to  improve  the  significant 
quality  attributes  including  security,  usability,  flexibility,  sustainability,  and  so  forth 

•  continue  process  improvements  by  identifying  more  places  to  use  tools  and  add  automation 

The  acquirer  should  encourage  and  actively  participate  in  improvements  that  enable  software  en¬ 
gineers  to  focus  on  requirements,  design,  and  analysis  activities  that  improve  the  quality  of  soft- 
ware-intensive  systems.  The  next  section  explores  tools  recommended  for  use  by  both  the  acquisi¬ 
tion  and  development  teams  that  can  help  focus  engineering  talent  to  create  quality  software. 
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2  Software  Development  Tools 


The  software  engineering  community  has  an  ever-increasing  set  of  tools  and  practices  aimed  at 
improving  the  quality  of  developed  software.  As  the  development  team  includes  these  tools  and 
processes  into  its  software  development  methodology  (coding,  unit  tests,  integration,  etc.),  it  helps 
to  improve  the  development  team’s  productivity,  while  at  the  same  time  it  gives  insight  and  con¬ 
fidence  to  the  acquisition  team.  This  section  gives  an  overview  of  tools  and  practices  available 
with  specific  details  on  the  tools  and  practices  most  effective  during  the  implementation  phase. 

2.1  Requirements,  Architecture,  and  Design  Overview 

It  is  beyond  the  scope  of  this  report  to  discuss  at  length  all  areas  of  requirements,  architecture 
(high-level  design),  and  detailed  design,  but  it  is  important  to  point  out  that  there  is  a  vast  array  of 
tools  and  practices  that  can  improve  the  effectiveness  of  the  development  team  during  these  activ¬ 
ities.  Many  of  these  methodologies  take  discipline  to  implement  and  execute,  but  as  described  in 
the  first  section,  the  goal  is  to  get  it  right  early  in  the  project.  Included  in  these  are 

•  Requirements:  Management  databases  to  aid  in  tracking  and  tracing  of  requirements  through¬ 
out  the  complete  development  life  cycle 

•  Software  Architecture:  Modeling  and  simulation  at  a  high  level  with  focus  on  the  quality  at¬ 
tributes  of  the  system.  These  visualization  and  simulation  techniques  enable  software  design¬ 
ers  to  consider  the  various  software  architectural  tradeoff  decisions  plus  can  aid  in  design  de¬ 
cisions  such  as  implementation  language,  SEI  COTS  Usage  Risk  Evaluation  methodology, 
reuse  of  code,  and  related  Integrated  Development  Environment  (IDE)  environments. 

The  above  activities  are  therefore  important  areas  in  which  the  acquisition  team  should  participate 
and  have  insight  into  design  and  process  decisions.  Many  formal  methodologies  (such  as  the 
SEI’s  Quality  Attribute  Workshop  and  Architecture  Tradeoff  Analysis  Method)  are  available  to 
help  teams  review  and  understand  the  architectural  and  design  decisions  that  are  made. 

2.2  Implementation 

As  pointed  out  in  many  software  development  models  and  textbooks,  writing  code  is  usually  a 
small  part  of  the  total  effort  needed  to  deliver  a  software  product  successfully.  Y et,  writing  code 
presents  the  easiest  opportunity  for  defects  to  be  introduced  into  the  product.  Many  automated 
tools  exist  to  instrument  code  in  order  to  catch  and  remove  defects  before  allowing  the  code  to 
move  into  integration  and  test.  An  important  tool  area  is  called  static  analysis,  which  often  in¬ 
cludes  code  beautifiers,  interface  extraction  (i.e.,  reengineering),  and  documentation  extraction. 
This  type  of  tool  supports  the  use  of  coding  standards  and  peer-review  processes  (to  be  covered  in 
the  next  section). 

2.2.1  Coding  Standards 

The  CERT  C  Secure  Coding  Standard  contains  a  good  description  of  a  coding  standard: 
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Coding  standards  encourage  programmers  to  follow  a  uniform  set  of  rules  and  guidelines  de¬ 
termined  by  the  requirements  of  the  project  and  organization,  rather  than  by  the  program¬ 
mer's  familiarity  or  preference.  Once  established,  these  standards  can  be  used  as  a  metric  to 
evaluate  source  code  (using  manual  or  automated  processes)  [CERT  Secure  Coding  Stand¬ 
ards  2011]. 

The  objectives  of  coding  standards  are  to  increase  the  maintainability,  reliability,  and  security  of 
the  code.  A  coding  standard  discourages  software  developers  from  using  practices  and  behaviors 
that  may  create  defective  code. 

A  coding  standard  usually  covers  these  four  areas: 

1.  Common  visualization:  includes  ways  to  have  common  layout,  code  indentation,  and  the 
usage  of  spaces 

2.  Naming  standards:  requires  developers  to  give  variables  and  functions  meaningful  names 
using  a  common  naming  convention  (e.g.  CamelCase,  under  score,  and  ALLCAPS) 

3.  Comment  standards:  give  guidance  as  to  how  many  and  what  types  of  comments  are  ex¬ 
pected 

4.  Coding  rales:  help  guide  the  programmer  as  to  the  proper  use  of  common  coding  patterns 
such  as  if,  switch,  do,  while,  variable  creation,  initialization,  and  assignment 

There  are  many  specific  features  and  automations  in  this  area;  some  of  them  include 

•  code  “beautifiers”  and  “pretty  printers”  to  enforce  layout  features  like  indentation,  aligmnent 
of  structures,  spacing,  matching  parentheses,  and  brackets 

•  comment  extractors  to  create  stand-alone  documentation  from  pre-formatted  comments  with¬ 
in  the  code 

•  static  analysis  enforcing  the  use  of  code  constructs  and  rules  from  the  coding  standard  includ¬ 
ing  naming  standard  and  comment  rales  in  addition  to  discourage  poor  coding  practices 

The  security  community  has  developed  coding  standards  such  as  Common  Weakness  Enumera¬ 
tion  (CWE)  and  the  aforementioned  CERT  C  Secure  Coding  Standard.  Other  communities  have 
created  similar  coding  standards  to  deal  with  safety,  performance,  and  maintenance.  These  are 
published  with  the  goal  of  making  the  set  of  coding  guidelines  and  best  practices  easily  available 
to  all  developers  so  they  can  incorporate  them  into  their  coding  standards.  The  rales  and  recom¬ 
mendations  in  these  standards  are  usually  based  upon  the  coding  language  and  how  it  is  used. 
They  define  common,  yet  poor  coding  patterns  that  developers  tend  to  use  that  affect  these  soft¬ 
ware  attributes. 

2.2.2  Static  Analysis 

Static  code  analysis  is  the  examination  of  aspects  of  the  software  system  implementation  that  are 
available  prior  to  execution.  The  concepts  of  the  implementation  language  are  brought  together 
with  the  coding  standard  as  the  metric  against  which  the  code  is  automatically  inspected.  Static 
analysis  tools  analyze  the  components  and  resources  of  an  application  without  having  to  ran  the 
application.  This  contrasts  with  dynamic  analysis,  which  requires  one  to  run  the  application.  Be- 
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cause  of  this  capability,  it  can  be  used  at  an  earlier  phase  of  the  development  cycle,  therefore  ex¬ 
posing  and  correcting  problems  before  entering  the  integration  and  test  phase. 

Static  analysis  can  be  viewed  as  an  automation  of  the  coding  standard  and  the  peer-review  pro¬ 
cess.  The  objective  of  these  tools  is  to  find  defects  at  machine  speeds  by  removing  the  burden  of 
manual  syntax  checking.  Static  analysis  reviews  compare  the  source  code  of  an  application 
against  a  set  of  coding  rules  to 

•  ensure  that  the  source  code  complies  with  the  selected  standards 

•  find  unwanted  dependencies 

•  ensure  that  the  intended  structural  design  of  the  code  is  maintained 

Static  analysis  is  a  mature  tool  type  that,  based  on  SEI  customer  engagements,  is  underutilized  in 
many  of  today’s  software  engineering  practices.  As  developers  find  ways  to  incorporate  the  use  of 
static  analysis  into  their  daily  development  processes,  it  brings  many  benefits: 

•  common  coding  patterns  that  lead  to  defects  are  found  almost  instantaneously 

•  gives  developers,  their  management,  and  the  acquirer  confidence  that  the  “simple”  defects  are 
found  early 

•  frees  up  valuable  software  engineering  time  to  focus  on  the  tougher  challenges 

A  common  comparison  in  the  industry  is  that  static  analysis  tools  should  be  treated  like  spell 
checkers;  human  intervention  is  required  to  find  the  important  problems.  Consequently,  static 
analysis  tools  will  not  answer  questions  such  as: 

•  Is  the  code  organized  so  that  another  developer  can  read  and  maintain  it? 

•  Is  it  a  robust  and  efficient  implementation  of  the  appropriate  algorithm? 

•  Does  the  code  work  as  required  and  documented? 

•  Are  the  unit  tests  sufficient  and  correct? 

When  static  analysis  is  used  as  part  of  a  software  development  process,  it  gives  the  development 
engineers  and  thus  the  acquirers  confidence  that  the  basic  coding,  design,  and  style  issues  have 
been  resolved  so  that  they  can  focus  on  these  more  difficult  questions. 

There  are  many  open  source  tools  along  with  commercial  static  analysis  tools  available  in  today’s 
market.  Each  has  various  features,  so  selecting  one  for  use  is  a  project  unto  itself.  Table  2  below 
contains  a  partial  list  with  brief  descriptions  of  defect  detection  that  can  be  accomplished  by  to¬ 
day’s  static  analysis  tools: 


Table  2:  Some  Types  of  Defect  Detection  Possible  With  Static  Analysis 


Defect 

Description 

Resolution 

Potential  null  dereference 

Code  attempts  to  call  a  method  or 
access  a  field  of  an  object  that  has 
been  set  to  null 

Check  for  failure  after  memory  allo¬ 
cation 

Unused  variable 

Variable  is  never  written  to  or  read 
from 

Remove  from  code  base 

Overly  broad  throw  of  exception 

Catching  any  exception  that  is  de- 

Understand  all  exception  handling 
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Defect 

Description 

Resolution 

rived  from  the  top-level  Exception 
class  but  treating  it  like  a  typical  non¬ 
runtime  exception  masks  runtime 
errors 

and  correct  using  best  practice 

Logging  warnings/errors  inconsist¬ 
ently 

Logging  is  not  consistent  throughout 

Pick  a  single  logging  mechanism 
and  use  it  exclusively 

Unchecked  method  return  values 

Returned  values  of  method  calls  are 
ignored 

Check  all  return  values 

Bypassing  of  architectural  or  securi¬ 
ty-based  API 

APIs  that  exist  to  enforce  consistent 
resource  allocation  or  input  valida¬ 
tion  are  not  used  in  all  situations 

Enforce  use  of  API  to  maintain  archi¬ 
tectural  and  security  integrity 

SQL  injection,  path  manipulation, 
command  injection,  log  forging 

When  querying  or  commanding  a 
database  or  operating  system,  user 
input  is  used  directly  to  form  the 
executed  instruction 

Use  parameterized  SQL  statements 
that  bind  parameters;  validate  and 
whitelist  all  input  used  in  dynamically 
created  instructions 

Unreleased  resource 

Calls  to  resource  allocations 
(memory,  threads,  etc.)  are  not 
matched  up  with  deallocation  calls 

Ensure  for  every  resource  used  that 
the  appropriate  management  meth¬ 
ods  are  used 

Use  of  dangerous  functions  related 
to  string  management 

Some  common  functions  exist  that 
cannot  be  used  safely  or  are  difficult 
to  use  safely 

Use  approved  functions  in  the  ap¬ 
proved  manner 

Debug  code  in  released  version 

Debug  code  can  affect  security, 
performance,  and  reliability  and 
should  be  removed  before  release  to 
the  user 

Remove  all  unit  test  and  diagnostic 
code  that  is  not  intended  for  ongoing 
product  support 

Empty  exception  catch  block 

Exceptions  not  handled  will  lead  to 
unexplained  software  failures 

Deal  with  all  expected  and  unex¬ 
pected  error  situations  making  sure 
the  information  is  available  to  diag¬ 
nose  failures 

Quite  often  today,  static  analysis  tools  also  include  features  that  help  developers  visualize  the  or¬ 
ganization  and  interfaces  of  the  major  components  of  the  software.  The  ability  to  analyze  and  vis¬ 
ualize  the  structural  and  architectural  features  of  the  developing  product  allows  for  early  analysis 
of  design  decisions  and  confidence  in  the  architectural  integrity.  In  addition,  these  tools  also  use 
various  methods  and  metrics  to  document  the  size  and  complexity  of  the  developing  software. 
This  allows  the  acquirer  and  developer  to  monitor  progress  and  detect  changes  that  will  impact 
program  cost  and  schedule. 

Some  of  the  criteria  to  look  for  when  deciding  on  what  tool  to  use 

•  built-in  rules  compatible  with  the  project’s  standards 

•  ease  of  managing  the  rales  (including  deleting,  creating,  and  reporting) 

•  ease  of  integration  with  other  software  development  tools  being  used 

•  automation  of  running  and  reporting  of  results 

2.3  Testing 

In  addition  to  the  implementation  phase  of  the  software  development  process,  static  analysis  and 
other  tools  are  necessary  to  integrate,  verify,  validate,  and  certify  the  delivered  system.  The  ac¬ 
quirer  needs  to  understand  how  these  processes  are  executed  as  the  results  are  quite  often  key  in¬ 
puts  to  the  acquirer’s  final  acceptance  process.  Integration  and  verification  (also  called  integration 


CMU/SEI-2013-TN-017  |  8 


and  test)  are  quite  often  planned  and  executed  by  the  same  team  of  engineers.  This  sometimes 
causes  confusion  as  these  are  separate  processes  with  a  different  set  of  goals. 

Integration  is  the  activity  after  coding  when  the  components  are  combined  to  make  a  working  sys¬ 
tem.  A  good  practice  here  is  to  build  up  the  system  step  by  step  from  only  a  few  components  to 
create  a  baseline  working  system.  The  ability  to  have  a  simpler,  working  baseline  version  aids  in 
identifying  defect  root  causes.  The  key  objective  of  this  process  is  to  make  sure  the  interfaces  that 
were  designed  on  paper  and  then  implemented  are  working  together.  If  a  formal  interface  tech¬ 
nique  like  IDL  for  Common  Object  Request  Broker  Architecture  (CORBA)  or  Simple  Object  Ac¬ 
cess  Protocol  (SOAP/HTML)  for  web  services  is  used,  then  there  are  static  analysis  tools  availa¬ 
ble  to  verify  the  actual  implementation  by  checking  that  the  interface  definitions  and  semantics 
are  consistent.  These  tools  define  the  interfaces  by  using  a  common  definition  language;  allowing 
the  components  to  be  built  automatically  with  specialized  tools,  they  can  compare  models  from 
the  design  phase  with  the  actual  implementation. 

Verification  quite  often  is  merely  considered  the  testing  phase  after  integration.  This  is  not  true. 
Verification  includes  the  testing  activities  from  the  start  of  development  and  includes  test  plan¬ 
ning,  test  procedure  development,  unit  testing,  test  automation  (or  not),  test  coverage,  system  test¬ 
ing,  and  more.  A  challenge  that  many  acquisition  programs  face  is  developing  testable  require¬ 
ments. 

Quite  often,  it  turns  out  that  the  verification  is  not  clean  and  leads  to  various  types  of  rework.  A 
common  practice  is  to  write  the  requirements  without  a  clear  idea  as  to  how  they  will  be  verified 
at  the  end  of  the  development  process.  By  using  automated  tests,  the  test  can  be  written  early  in 
the  development  process  to  help  define  the  requirement  and  allow  it  to  be  mn  frequently  during 
the  development,  integration,  and  test  phases. 

The  dynamic  analysis  tools  discussed  below  are  used  in  both  integration  and  verification.  Auto¬ 
mation  (discussed  further  in  section  3.1)  is  usually  developed  and  debugged  for  initial  use  during 
the  integration  stages  of  the  project  and  will  be  further  employed  during  verification  and  possibly 
the  certification  activities.  A  key  benefit  to  the  acquirer  is  that  these  same  automated  tests  become 
valuable  as  regression  test  suites  during  the  sustaimnent  phase  of  the  product  life  cycle. 

2.3.1  Dynamic  Analysis 

Dynamic  analysis  is  an  important  type  of  tool  usually  used  during  integration  and  verification  that 
helps  to  check  for  various  qualities  of  the  code  such  as  performance,  security,  and  availability. 

The  use  of  dynamic  analysis  must  be  planned  for  from  the  beginning  of  the  design  phase.  Dynam¬ 
ic  analysis  uses  data  gathered  from  additional  instrumentation  code  whose  purpose  is  to  capture 
the  events  in  real  time.  While  the  capability  needs  to  be  included  early  in  the  design  process,  the 
system  needs  to  be  sufficiently  integrated  for  dynamic  analysis  to  be  useful.  Thus,  defects  are 
found  later  in  the  process  than  those  discovered  by  static  analysis  tools.  Dynamic  analysis  checks 
for 

•  performance  degradation 

•  memory  management  issues  like  leaks,  bad  allocations,  and  memory  corruption 

•  threading  and  interlock  conditions  that  can  impact  performance  or  cause  run-time  failures 
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•  race  and  deadlock  conditions 

•  security  vulnerabilities 

•  code  coverage  monitoring 

These  are  important  features  to  the  acquirer,  and  the  use  of  dynamic  analysis  should  be  planned 
for  early  in  the  system  design  phase  even  though  the  actual  analysis  does  not  occur  until  late  in 
the  development  cycle. 

2.3.2  Use  of  Static  Analysis  in  Certification 

Quite  often,  large  acquisition  programs  face  certification  challenges  by  an  independent  organiza¬ 
tion.  This  accreditation  or  authorization  to  be  utilized  for  the  intended  purpose  happens  frequently 
in  systems  intended  for  government,  military,  medical,  and  financial  use.  Certification  is  an  ap¬ 
proval  process  for  the  system  or  parts  of  the  product,  usually  with  a  strong  focus  on  security. 

These  processes  typically  involve  checking  that  the  code  meets  selected  coding  standards  either 
through  manual  or  automated  means. 

A  key  activity  of  certification  usually  involves  independent  static  analysis  of  the  code  base.  If  this 
analysis  is  executed  against  a  different  coding  standard  or  using  a  different  set  of  static  analysis 
rules  and  patterns  than  the  developer  used,  it  will  tend  to  show  a  large  number  of  potential  defects 
— many  of  which  will  be  later  identified  as  false  positives.  For  each  positive,  the  developer  and 
certification  authority  must  then  negotiate  to  detennine  whether  the  flagged  code  is  in  fact  in 
compliance  (false  positive),  may  remain  in  place  as  a  documented  deviation,  or  requires  repair. 
According  to  CERT’s  recent  Source  Code  Analysis  Laboratory  (SCALe)  for  Energy >  Deliveiy  Sys¬ 
tems  report, 

Deviations  should  only  be  used  infrequently  because  it  is  almost  always  easier  to  fix  a  cod¬ 
ing  error  than  it  is  to  provide  an  argument  that  the  coding  error  does  not  result  in  vulnerabil¬ 
ity.  ...  Depending  on  the  analyzers  used,  it  is  not  uncommon  for  code  bases  to  have  substan¬ 
tial  numbers  of  false  positives  in  addition  to  the  true  positives  that  caused  the  software  to  fail 
conformance  testing.  False  positives  must  be  eliminated  before  a  software  system  can  be  cer¬ 
tified.  Flowever,  analyzing  the  code  to  determine  which  diagnostics  are  false  positives  is  time 
consuming  and  labor  intensive.  Furthermore,  this  process  needs  to  be  repeated  each  time  the 
code  base  is  submitted  for  analysis.  Consequently,  preventing  the  issuance  of  diagnostics  de¬ 
termined  to  be  false  positives  can  reduce  the  cost  and  time  required  for  conformance  testing 
in  most  cases  [Seacord  2010]. 

The  ongoing  challenge  by  the  acquisition  team  is  to  continually  certify  new  releases  of  the  code. 

In  attempting  to  obtain  certification  in  a  timely  fashion,  the  certifying  organization  sometimes 
participates  early  in  the  acquisition  process.  Acquirers,  security,  certification,  and  test  teams 
should  make  sure  the  right  coding  standards  and  specifications  are  part  of  the  contract  and  en¬ 
forced  by  the  software  developer  on  a  daily  basis  by  using  appropriate  static  analysis  tools  along 
with  other  system  engineering  practices. 
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3  Overarching  Software  Development  Practices 


When  considering  many  of  the  various  software  development  process  areas,  mature  tools  and 
practices  with  proven  techniques  are  quite  often  available.  Executing  the  process  areas  well  across 
the  whole  development  life  cycle  is  important  to  the  timely  delivery  of  quality  software.  However, 
for  the  purpose  of  this  technical  note,  we  focus  on  two  practices — automation  and  peer  review, 
followed  by  a  quick  look  at  defect  management  and  quality  assurance  that  support  the  automation 
and  peer  review  practices. 

Quite  frequently  automation  and  peer  review  practices  are  skipped  or  deferred  during  the  devel¬ 
opment  process  in  an  attempt  to  meet  cost  and  schedule.  As  these  practices  have  been  widely  rec¬ 
ognized  over  the  years  to  pay  off  in  terms  of  quality,  they  should  not  be  dismissed  lightly.  From 
the  acquirer’s  point  of  view,  automation  and  peer  review — when  executed  appropriately — can 
supply  confidence-building  insight  into  the  acquisition. 

3.1  Challenges  of  Automation 

Christie  et  al  define  automation  as  “. .  .computer-based  support  for  the  flow  of  work  between  indi¬ 
vidual  tasks.  Processes  (large  or  small)  are  said  to  be  automated  if  manual  control  of  task  initia¬ 
tion  or  sequencing  is  transferred  to  the  computer  [Christie  1996].”  The  implementation  and  com¬ 
plexity  of  automation  changes  over  time  due  to  new  and  improved  tools  and  techniques.  But  in 
effect,  software  development  and  test  automation  tools  perform  what  were  previously  manual 
processes  and  allows  them  to  be  completed  without  human  intervention. 

The  concept  of  automation  can  be  applied  across  the  implementation,  integration,  and  test  pro¬ 
cesses  used  to  develop  a  software  product.  COTS  tools  are  available  that  can  quite  often  easily  be 
incorporated  into  most  development  processes.  Common  automations  include 

•  automated  test  frameworks  or  test  harnesses  which  enable  developers  to  test  the  code  from 
unit  to  integration  testing 

•  user  interface  automation  to  enable  the  repetition  of  user  text  and  mouse  input  along  with  ver¬ 
ification  of  the  expected  events  from  each  input 

•  COTS  products  geared  toward  online  application  testing 

Planning  for  the  use  of  automation  early  in  the  life  cycle  brings  several  key  advantages  to  both  the 
developer  and  acquirer: 

•  Repeatability  of  testing  allows  for  consistent  regression  testing,  leading  to  confidence  and 
predictability  in  early  and  operational  releases. 

•  Automated  frameworks  can  enable  the  development  and/or  sustaimnent  team  to  add  new  fea¬ 
tures  efficiently  and  simplify  the  remediation  of  defects. 


CMU/SEI-2013-TN-017  |  11 


Deciding  whether  and  how  to  implement  automation  tools  and  techniques  is  as  important  as  im¬ 
plementation  decisions  within  the  product  itself.  Although  there  are  many  opportunities  to  use 
automation  within  software  development  processes,  the  decision  to  automate  a  particular  technical 
process  usually  involves  significant  tradeoff  analysis.  The  automation  process  alone  creates  the 
need  for  the  development  organization  to  develop  expertise  in  the  use  of  any  automation  tools 
they  employ,  requiring  staff  training  and  possibly  the  hiring  of  new  talent.  In  addition,  creating 
and  maintaining  the  automation  can  become  a  project  unto  itself.  Maintaining  automation  scripts 
and  software  has  the  same  types  of  challenges  associated  with  maintaining  other  software  prod¬ 
ucts  and  therefore  must  be  considered  when  deciding  to  automate.  Automation  tools  themselves 
necessitate  requirements,  design,  and  documentation  with  the  challenge  to  keep  the  code  updated 
and  documented  accurately.  Quite  often,  automation  objectives  also  require  a  make-versus-buy 
decision  with  all  of  the  associated  tradeoffs  to  be  considered. 

Automated  tools  are  extremely  important  in  creating  a  high-tempo  development  process.  Auto¬ 
mated  check-in  and  analysis  of  code  bases  can  quickly  point  to  where  changes  have  caused  issues, 
and  attention  can  be  directed  more  readily  to  those  areas.  After  incorporation  of  basic  automation, 
further  concepts  are  worthy  of  pursuit.  Automated  analysis  of  higher  order  issues  associated  with 
component  interaction,  interface  dynamics,  and  model-driven  architecture  are  some  examples.  In 
most  of  today’s  development  methodologies  using  iterative  approaches,  the  ability  to  automate  is 
considered  a  necessity  for  success.  When  automation  is  planned  for  and  applied  early,  the  benefits 
ripple  across  the  product’s  life  cycle. 

3.1.1  Automatic  Code  Generation 

Automatic  code  generation  is  used  in  combination  with  high-  and/or  detailed-level  design  and 
modeling  tools.  Once  the  design  is  finalized,  the  code  is  generated  using  the  design  and  imple¬ 
mentation  details  supplied  by  the  design  tool.  In  the  case  of  a  high-level  design  tool,  the  skeleton 
or  outline  of  the  code  is  automatically  generated  to  allow  manual  creation  of  the  actual  functions 
or  methods.  Depending  on  the  level  of  detail  in  the  design  or  modeling  tool,  the  actual  software 
for  the  functions  or  methods  may  in  fact  be  created. 

Many  of  the  activities  that  a  programmer  does  while  writing  code  offer  opportunities  for  automa¬ 
tion.  Detailed  design  tools  allow  the  developer  to  specify  the  software’s  parameters  and  common 
algorithms  in  a  format  that  is  easily  viewed  and  maintained.  This  allows  a  code  generation  tool  to 
produce  the  code. 

A  key  consideration  when  generating  code  is  how  that  code  will  be  sustained.  The  generated  code 
can  be  considered  a  starting  point  for  manual  programmer  modifications  such  as  creating  a 
framework  of  code  modules  from  an  initial  high-level  design.  A  better  practice  is  to  use  the  design 
or  modeling  tool  to  make  changes  which  requires  maintenance  of  the  tool  and  the  associated  mod¬ 
el  throughout  the  sustainment  period.  Another  good  practice  is  to  clearly  identify  how  the  gener¬ 
ated  code  was  written  to  enable  future  evaluators  and  sustainers  to  make  effective  decisions. 

Automatic  code  generation  enables  the  developer  to  easily  employ  a  defined,  repeatable  process 
in  a  consistent  coding  style.  This  gives  the  acquirer  confidence  that  development  is  done  in  a  re¬ 
peatable  fashion,  following  good  programming  practices  that  are  geared  toward  preventing  de¬ 
fects. 
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3.1.2  Continuous  Integration  and  Test 


Continuous  integration  can  be  effective  for  almost  all  types  of  software  developments.  Continuous 
integration  is  enabled  by  using  a  configuration  management  system  with  the  ability  to  automate 
the  build  process.  Successfully  built  code  is  then  automatically  tested  to  verify  that  the  newly  built 
product  works  correctly.  Automating  the  build  and  unit  test  environment  is  common  in  many  of 
today’s  software  development  processes.  This  enables  daily  build  and  integration  along  with  au¬ 
tomated  static  analysis  scans  and  unit  tests.  A  greater  level  of  automation  enables  quick  discovery 
of  defects  and  enables  the  removal  of  defects  closer  to  the  insertion  point. 

With  some  scripting,  the  code  can  be  generated,  compiled,  statically  analyzed,  unit  tested,  and 
delivered  into  an  integration  environment  automatically  within  hours.  Errors  found  during  these 
steps  can  be  sent  to  the  appropriate  developer  for  immediate  action.  With  this  kind  of  response, 
the  time  between  implementation  and  system  testing  can  be  made  extremely  short  so  that  devel¬ 
opers  and  testers  can  focus  on  their  primary  activities.  The  development  team’s  goal  should  be  to 
improve  this  process  so  that  defects  are  found  and  removed  as  soon  after  writing  the  code  as  pos¬ 
sible. 

3.1.3  Test  Automation 

Instituting  test  automation  enables  the  developer  to  shorten  implementation  and  system  test  cy¬ 
cles,  giving  the  acquirer  greater  insight  through  the  automatic  collection  of  accurate  and  revealing 
feedback  into  the  state  of  the  developing  product.  As  described  in  previous  sections,  automation 
of  the  static  analysis  and  unit  testing  can  easily  be  built  into  the  development  process.  Tools  also 
exist  to  enable  automation  at  a  higher  level  so  that  regression  and  functional  testing  can  be  done 
on  a  frequent  basis.  The  ability  to  create  a  test  that  is  repeatable  without  human  intervention  al¬ 
lows  both  the  developer  and  acquirer  to  gain  confidence  in  the  system  more  quickly.  As  described 
in  the  previous  section,  automation  does  not  come  for  free.  The  ongoing  maintenance  of  an  auto¬ 
mated  test  framework  can  be  a  project  unto  itself.  There  are  many  books,  courses,  and  tools  avail¬ 
able  on  this  subject. 

3.2  Peer  Review 

For  the  purposes  of  this  technical  note,  the  authors  chose  a  broad  definition  of  peer  reviews.  Peers 
include  not  only  the  technical  team  members  but  also  include  stakeholders  such  as  the  acquisition 
team  members  and  the  end  users.  Widening  of  the  peer  review  concept  beyond  the  bounds  of  the 
technical  team  encourages  earlier  looks  by  acquirers  and  customers/end  users  so  functional  and 
user  experiences  can  be  dealt  with  sooner. 

Peer  review  is  the  evaluation  of  creative  work  or  performance  by  other  people  in  the  same 
field  in  order  to  maintain  or  enhance  the  quality  of  the  work  or  performance  in  that  field. 

ft  is  based  on  the  concept  that  a  larger  and  more  diverse  group  of  people  will  usually  find 
more  weaknesses  and  errors  in  a  work  or  performance  and  will  be  able  to  make  a  more  im¬ 
partial  evaluation  of  it  than  will  just  the  person  or  group  responsible  for  creating  the  work  or 
performance  [LINFO.org  2005], 

Participation  by  the  acquirer  in  peer  reviews  allows  for  deeper  insight  into 
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•  the  development  team’s  level  of  understanding  of  the  requirements 

•  the  design  decisions  that  are  made  by  the  development  team 

•  the  quality  and  sustainability  of  the  developed  software  early  in  the  life  cycle 

•  coverage  and  depth  of  the  planned  verification  activities 

Peer  review  methodologies  encourage  getting  more  eyes  looking  for  problems.  As  discussed  in 
section  2.1,  various  methodologies  can  be  used  throughout  the  development  life  cycle  to  review 
the  architecture,  design,  and  developed  code.  These  techniquesinclude  (but  are  not  limited  to) 
formal  inspections,  walkthroughs,  and  pair  programming.  The  concept  of  peer  reviews  is  to  geta 
few  people  in  a  room  to  focus  on  a  small  section  of  code.  Today  there  are  also  some  workflow 
management  tools  that  allow  geographically  or  time-dispersed  teams  to  hold  peer  reviews. 

These  methods  are  great  ways  to  share  the  knowledge  and  increase  communication  between  the 
acquisition  and  development  teams.  Each  of  these  methods  has  benefits  and  weaknesses  and  relies 
on  the  team’s  communication  and  social  capabilities.  The  use  of  static  analysis  tools  and  code 
beautifiers  before  the  peer-review  process  begins  will  help  the  group  focus  on  the  tough  require¬ 
ments,  design,  implementation,  and  defect  resolution  issues. 

3.3  Common  Software  Development  Processes 

The  tools  and  practices  in  addition  to  automation  techniques  discussed  earlier  in  the  paper  are  ef¬ 
fective  additions  to  many  common  software  engineering  processes.  This  section  explores  how 
they  can  be  incorporated  into  configuration  management  and  quality  assurance. 

3.3.1  Configuration  Management 

Configuration  management  (CM)  is  a  very  mature  area  in  which  the  market  offers  a  wide  variety 
of  solutions  ranging  from  those  that  may  be  free,  open  source,  and/or  commercially  marketed. 
Acquirers  should  make  sure  their  development  organizations  integrate  these  tools  into  their  daily 
work  processes  to  manage  the  identification  and  control  of  the  software  being  created.  Some  of 
the  interesting  measures  that  acquirers  should  consider  reviewing  are  frequency  of  check-ins, 
software  size,  and  version  information.  Proper  use  of  these  tools  along  with  the  appropriate  re¬ 
ports  is  a  basic  confidence  builder  for  the  acquisition  team  that  the  developer  is  doing  this  most 
basic  software  engineering  process  appropriately. 

One  of  the  more  recent  features  available  in  CM  tools  is  the  automation  of  steps  such  as  check-ins 
and  other  workflow  elements  (e.g.,  initiating  cycles  of  static  analysis,  compiling,  and  builds  upon 
check  in).  A  challenge  in  making  automation  successful  is  to  use  a  CM  tool  that  makes  the  im¬ 
plementation  of  this  automation  simple  and  effective  in  the  contractor’s  development  environ¬ 
ment,  which  may  mean  seeking  out  easy-to-use  interfaces,  instantaneous  check-in  times,  and  ef¬ 
fective  remote  access. 

The  acquisition  team  relies  on  the  developer  to  create  a  CM  plan  to  manage  the  software-related 
work  products  in  addition  to  developing  code  according  to  the  plan.  Furthermore,  remember  that 
acquirers  also  bear  responsibility  for  CM  on  programmatic  documents.  To  enable  the  acquisition 
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team  to  participate  effectively  in  peer  reviews  of  the  developing  system,  appropriate  remote  ac¬ 
cess  should  be  easy  and  accessible  as  the  product  evolves. 

A  jointly  agreed  upon  CM  plan  including  use  of  automation  features  and  easy  access  by  both  par¬ 
ties  is  important  to  gain  efficient  and  effective  interaction  between  the  developer  and  acquirer.  For 
example,  such  a  plan  enables  a  program  manager  to  understand  how  the  CM  system  maintains 
linkages  between  technical  and  programmatic  documentation 

3.3.2  Defect  Management 

Defect  management  processes  help  answer  some  of  the  most  common  questions  that  acquisition 
teams  ask:  What  is  the  quality  of  the  product  and  when  will  it  be  finished?  Tools  range  from 
homegrown  databases  to  integrated  commercial  off-the-shelf  (COTS)  products  for  tracking  de¬ 
fects  from  start  of  development,  through  deployment,  and  into  sustainment.  In  addition  to  manag¬ 
ing  defects  on  a  daily  basis,  these  tools  help  gather  measurements  that  help  the  development  and 
acquisition  teams  gauge  the  quality  of  the  developing  product.  Data  such  as  defect  density  and 
defect  detection/removal  efficiency  are  important  for  knowing  the  quality  of  the  system  and  for 
deciding  when  to  release  it. 

This  class  of  tools  enable  wot  cause  analysis,  which  is  a  key  activity  that  requires  the  analytical 
capabilities  of  the  technical  skills  of  all  the  team  members.  Root  cause  analysis  seeks  to  identify 
the  source  of  a  problem  so  that  classes  of  defects  can  be  found  and  corrected  in  both  technical  and 
process-related  areas.  Proper  use  of  defect  management  tools  will  help  to  minimize  the  engineer¬ 
ing  effort  needed  to  gather  information,  measure,  and  report  patterns  in  defects. 

3.3.3  Quality  Assurance 

Quality  Assurance  (QA)  takes  many  forms  from  company  to  company.  An  important  element 
from  the  acquirer’s  point  of  view  is  that  the  QA  processes  and  methodologies  can  help  the  acquir¬ 
er  gain  insight  into  the  developer’s  progress.  QA  monitors  the  adherence  to  the  defined  develop¬ 
ment  process  in  addition  to  quality  checks  of  the  developed  artifacts,  supporting  the  acquisition 
team’s  goal  of  a  quality  product.  Since  the  acquisition  team  has  ultimate  responsibility  for  what  is 
being  developed,  they  should  witness  key  development  team  QA  activities  to  gain  confidence  that 
the  development  team  is  working  to  the  planned  development  process.  Activities  can  include 

•  observation  and  participation  in  peer  review  and  evaluation  activities 

•  observation  of  development  team  verification  and  validation  activities 

•  observation  and  review  of  process  compliance  reviews 

In  addition,  the  acquisition  team  can  gain  confidence  that  many  of  these  basic  software  engineer¬ 
ing  processes  are  well-instituted  if  the  tools  are  used  properly  on  a  regular  basis.  Several  examples 
include  the  following: 

•  requirements  management  repositories  -  quickly  check  and  visualize  the  tracing  between  par¬ 
ent  and  child  requirements  along  with  tracing  to  design  and  test  elements 

•  static  analysis  —  constantly  enforce  the  coding  standard 
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•  configuration  management  —  quickly  check  and  visualize  baselines  for  all  types  of  design  arti¬ 
facts  from  requirements  to  design  to  code  to  test 

All  of  the  tools  and  practices  discussed  in  this  technical  note,  including  automation,  enable  QA 
and  the  acquisition  team  to  gain  insight  quickly,  thus  allowing  time  to  focus  on  design  and  deci¬ 
sion  activities  over  rote  tasks. 
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4  Summary  and  Concepts  for  Further  Consideration 


The  use  of  the  tools  and  practices  discussed  in  this  technical  note  should  be  expected  and  encour¬ 
aged  by  the  acquisition  team  as  they  are  mature,  common  tools  used  in  today’s  software  develop¬ 
ment.  As  such,  the  tools  supply  key  insights  into  the  quality  of  the  developing  product  along  with 
indicators  as  to  when  the  product  will  be  completed. 

4.1  Justification  for  Improvement 

Unfortunately,  quite  frequently  the  tools  and  practices  discussed  in  this  technical  note  are  consid¬ 
ered  extra  and  unnecessary  when  schedules  are  tight  and  limited  funding  is  available.  There  are 
budget  and  schedule  considerations  to  using  these  tools  and  practices  such  as  tool  costs  including 
the  number  of  licenses,  ongoing  maintenance  of  the  tool,  and  training  costs  over  the  whole  soft¬ 
ware  development  cycle.  In  addition,  the  dedicated  time  to  execute  many  of  these  practices  and  to 
manage  these  tools  is  also  perceived  as  an  ineffective  use  of  costs  and  schedule. 

Improved  quality  is  sometimes  hard  to  measure,  but  it  tends  to  manifest  in  less  effort  needed  due 
to 

•  deliver  the  product  more  predictably,  which  leads  to  lower  development  cost  and  schedule 

•  reduced  defects  in  the  released  product,  thus  leading  to  lower  maintenance  costs 

•  improved  development  processes,  leading  to  lower  enhancement  costs  through  the  ability  to 
add  new  features  more  easily 

4.2  Acquisition  Strategy  and  Adapting  to  Change 

A  common  challenge  to  large  projects  and  programs  is  that  they  usually  last  many  years.  This  fact 
alone  is  likely  to  guarantee  that  requirements,  people,  tools,  and  practices  will  change  multiple 
times  during  the  development  effort.  If  the  acquisition  strategy  does  not  account  for  adapting  to 
change,  then  it  is  setting  up  the  acquisition  for  failure.  While  there  is  the  feeling  of  safety  by  keep¬ 
ing  the  development  environment  constant,  not  upgrading  the  tools  and  practices  in  a  protracted 
project  forces  the  development  team  to  work  with  older,  less  effective  tools  on  what  is  typically  a 
much  more  complicated  system. 

Why  are  opportunities  to  improve  key  to  process  improvement?  It  is  the  planning  to  improve 
throughout  the  project  that  enables  the  benefits  of  process  improvement;  chiefly,  improved  quality 
of  the  final  product.  Process  improvement  can  take  place  only  when  tools  and  practices  are  al¬ 
lowed  to  change.  If  the  change  cycle  takes  years  for  each  modification,  then  very  little  improve¬ 
ment  can  happen.  This  means  that  the  acquirer  should  find  an  acquisition  strategy  that  allows  for 
development  tools  and  practices  to  occur  frequently  while  measuring  the  results. 

Consider 

•  frequent  update  of  COTS  tools  to  keep  them  current  and  allow  the  use  of  the  latest  efficiency 
features 
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•  ongoing  sustainment  of  in-house  developed  tools  to  keep  them  useful 

•  training  for  new  members  of  the  acquisition  and  developer  team  in  existing  tools  and  practic¬ 
es 

•  updating  and  the  associated  training  for  new  tools  and  practices 

4.3  Acquirers  and  Developers 

As  an  acquirer,  instead  of  asking  your  development  team  typical  status-related  questions  such  as 
’’When  will  you  be  done?,”  the  authors  recommend  trying  some  of  these  questions: 

•  What  is  painful  about  the  current  software  development  process? 

•  What  code  quality  improvement  suggestions  have  you  made  recently? 

•  What  automation  and  peer-review  improvements  are  you  piloting? 

•  Is  quality  assurance  verifying  adherence  to  previously  agreed  upon  processes? 

Consider  these  simple  example  strategies: 

•  Look  for  developers  to  use  static  analysis  on  code  check-in  as  part  of  the  software  develop¬ 
ment  process,  and  explain  the  types  of  analysis  that  they  perfonn.  This  process  can  be  auto¬ 
mated  so  that  the  impact  is  minimal  and  bugs  are  dealt  with  before  unit  and  regression-level 
testing. 

•  The  use  of  static  and  dynamic  analysis  tools  as  part  of  the  development  process  can  provide 
acquisition  and  certification  organizations  with  key  insights.  A  close  relationship  with  the  de¬ 
velopment  team  can  allow  the  acquirer  and  certifier  the  opportunity  to  encourage  the  use  of 
appropriate  coding  and  testing  standards  and  practices  on  a  continual  basis.  The  execution  of 
these  processes  can  then  be  audited  and  inspected  to  gain  confidence  and  certify  that  the 
software  meets  some  subset  of  the  required  standards.  The  risk  of  failing  certification  at  the 
end  is  greatly  reduced. 

•  The  acquisition  team  should  receive  frequent,  interim  integration  and  test  reports  from  the 
development  team  to  provide  confidence  in  the  expected  quality,  cost,  and  schedule.  A  formal 
report  that  is  completed  at  the  time  of  delivery  can  be  used  for  the  certification  process.  This 
should  be  a  non-event  as  the  interim  reporting  process  will  discover  problems  early  in  devel¬ 
opment. 

Usually  the  contractor  has  a  variety  of  tools  and  practices  in  their  toolkit  already — backed  up  by 
corporate  investment  enabling  deep  experience  and  knowledge.  An  acquirer  would  do  well  to 
consider  asking  questions  about  where  in  the  life  cycle  the  developer  employs  the  available  tech¬ 
niques  and  tools.  While  the  acquirer  is  rarely  in  a  position  to  dictate  about  the  tools  and  practices 
used,  just  asking  the  question  and  setting  expectations  helps  to  motivate  the  acquirer  into  using 
and  enabling  them. 

This  paper  gives  to  both  the  acquirer  and  developer  a  better  understanding  of  the  benefits  of  using 
these  tools  and  practices.  There  should  be  obvious  justification  now  for  the  added  cost  and  effort 
in  training.  The  success  of  using  these  tools  requires  close  involvement  between  the  acquirer  and 
developer  organizations.  In  the  past,  insight  by  the  acquisition  team  was  focused  at  key  mile- 
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stones.  The  use  of  automation  and  analysis  tools  and  practices  today  allows  for  a  collaborative 
environment,  which  reduces  the  risk,  conflict,  and  rework  that  has  plagued  software  developments 
in  the  past. 
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test  automation,  and  peer  review  techniques.  In  addition  to  the  aforementioned  tools  being  important  for  software  developers,  they  offer 
significant  insight  and  confidence  to  customers  who  acquire  software  products. 

There  are  many  benefits  from  making  these  tools  and  practices  integral  to  software  development  and  acquisition  processes,  including: 

•  reduced  risk:  programs  deliver  more  predictably 

•  improved  customer  satisfaction:  products  developed  experience  fewerfield  defects 

•  lower  cost  of  ownership:  programs  experience  lower  life-cycle  maintenance  costs  when  de-ployed  operationally 

This  technical  note  provides  an  introduction  to  key  automation  and  analysis  techniques,  the  use  of  which  the  authors  contend  will  benefit 
motivated  acquirers  and  developers. 
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